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InlrodiH'lion. The central theme of this study la the classification of 
<:rrl.iLLCi nix>rru!trii:;il [jTii]it;r1its ;.i r.:nlin : -: I:. !.hi; lv|., ,:l fomputfttiOIl 
necessary to determine whether S #iv«n ^un; h?*$ them, Consider* for 
eiample, the following algorithm to determine whetiter a figure X u 
cvruvx. For each pair of points, ip.q) we define the function 

tpfiX} — 1 if ( p E X and q<E.X and midpoint ( p , q) {£ X ) 

= OEh*|-w5H;, 

Then A' is convex if and only if no i K {X) = 1 for any pair of point* [p,q). 
This shows, in, a sense we shall presently define more precisely, that 
the "jdubal" property of eonveiity con be determined by a simple compu- 
tation from the "local" properties ^. 2 Thus, if d^^Jt} means that 
"X is convei," we have 

Now we generalise this. We wiy that a property $ is fl/" Ki/^r If if k is the 
smallest integer for which there eiuta a family + of predJeuluK. vavh of 

which depends only on a subset of t points of the figure X r and real 
numbers a^ towjcijiti'd whli h:;i<::i mpmljer # of + such that 






In thia sense, we can assert that the order a/ f mM i$ at most it. The de- 
termination of the orders of simple gen-metrical properties tuint out to 
be far from trivial and presents many surprises, in fact, tliK Ri^al^r \\;i- j 
of the following utuuysit seems to be needed to prove that coanectedttesi 



1 Ttila *Wi wsa mjppOtWC. ill part hy PttjJetC MAC, and M.l.'P. meoidi pto>Hi ipnfutfp«l 
by ihf Advanwd Rtrairch PrftjMtt Aparey, DapjtlrrvwH of Dtttaiw, under UEce of MnvnL 
H«Mrch conlrncti; NonrilOQiOl) and (02). 

Wf iikniiify "ppjiiHrtv" Cur "jwwlrt*!.*''! with T-h* cbHr&r Lima tic fcnclinn nJ the jul pf 
objrcls r .hai have clic praprrty. 

ir* 
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i* not of any finite- ■older,. Le„ 

7*he re isnok fay cisfiicA C0JwecJl«iWss- z> vfwikr L 

This remains true if we relax the definitions,, as we shall, to make the 
sums finite, by considering the plane m u fine-grained Ignite chess-board, 
considering it* secures to Lie points, and allowing only figures which contain 
each square entirety w not at ah — that ii, wo will consider a di¥<LTe:c 
model of geometry. 

Apart from the purely mathematical interest of the remits that mint 
from it, we consider the concept of finite order worthy of study for a number 
of reasons connected with the theories of oumputation and of pattern 
recognition, We ahalL briefly outline Eome of these reasons. 

Motivation af This Study. 

(aj Lacai us. glaboi ^■ami-tru.- properties. In problems of geometric pattern 
r:^-.!,:nil n: n. ,,-C- i> .(:1 :i: ;,;:>:: Ciwral excenr C:-.:: :.rn u«C "' k>C;il" priip-rLn-!i 
evidence obtained from, looking at small portion* of an object— as a hasis 
for judgements about the "global" -character of the. object. For example, 
one can distinguish "line n drawings from other pictures on the basis of the 
existence of no interior points in the draw intf— and this can be determined 
by a simoLe wmbination of evidence obtained frum eiamining arbitrary 
small nejahhorhooua, On the other hand, one cannot obtain " local" evidence 
in favor of a drawing being "connected"— or so one might suspect- without 
having to combine such collections of evidence by a very complicated 
procedure. 

Our first attempt to study this waa baaed on ibe idea of diartteteT'Tvatrict&i 
predicate.^ is, t the lestrictLfua on the "local" properties lb on the diame1«r 
of the set of points on which the.v depend rather than the number of point*. 
The results of this study are summarised in §IX. However it soon bEeame 
dear that the more interesting- concept is order- restriction, and that the 
distinction we were seeking waa not so much a question of geometry as a 
Question about the theory of computatioft. 

£b> Serial Us. parattet ccmpukUivn. What rhmrueterLzes the extent m 
which bo. algorithm can have an essentially wra( F as opposed to paraUei, 
character? That is, to what degree can a computation he *ped-up by doing 
several subcomputations, at the earns- time? One would suspect, for example, 
that in manv successive appiojcirimlion computation* there is little to bo 
gained except, at great eipense and rfiilundancy, by parallel processing, 
We were led to suppose that the same is true for geometric Connectedness 
recognition, One way to regime that a set is disconnected {connected} 
is to find that there is a (in.i) eurve dividing the set without intersecting it, 
One could therefore examine in parallel all possible separating curves. 
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rathet- th*n serially trace through the paths within the $et. Rut it would 
seem that the price of speeding-up the computation this, way is superbly 
costly, and one loots frV « way to gut theoretical estimates, of what is the 
exchange rate between the minima] serial and parallel amounts of corn [Nu- 
tation. {The goal must he an exchange-cost curve.* One might hope that 
study of a particular problem, e.g., connectedness., would yield some insight 
into this general question of computation complexity for finite probLems. 
comparable with r say, that achieved in the theory of complexity of the 
recursive [unctions (Blum)- 

(cj Theory of perceptrQiis und linear separability. The pattern- recognition 
achemB known aa the percepirvn (Rosenblatt |6]) is known to he capable 
of learning <o make- any pattern discrimination which it within the scope 
of its potential ability — that is, if there is a set of parameter values that 
wild suffice, it will find them, ThuH, a good deal is known ahouL this ayalein's 
learning ability, and therefor* one m particularly intareated. to know what 
is the scope of potential ability. Curiously enough, there seems to he nothing 
in the Litrifc perception literature on this question, and the present paper 
stems to he the fir^t to link the linear-separation problem with the jjeometric- 
property problem. 

The perceptran (and iti derivatives-) are uf considerable interest mathe- 
matically because they are perhaps the simplest nontrlvial parallel 
machines. One therefore rtutfht U* understand them thoroughly — as a sort 
of " linear case"- if one is to get any satisfactory theory of "higher-order" 
parallel ci imputation schemes. 

(d) Mnihrnuiiiwil asptxtx. Linear separation computations have con" 
siderable mathemiitkal significance In themselves. For example, if we 
ask for a maximum likelihood decision process, based on Bayegian u$e of 
the results of statistically independent experiments, one obtains (Miosltv 
andSelfridge fs]) a linear separation procedure. For another example, the 
generalization (in Jl} of Boolean disjunctive normal form appears to yield 
surprising rmd fruitful results, Finally, the combination of group theory and 
linear mequaUtiea seem* to promise some nsw combinatorial tecl»iii|iu*v 

1. Theory of linear Bwlean separation functions. Jn this section we shall 
confine ourselves to the analysis pf the linear representation of predicate* 
defined on an abstract sb6 R 7 without any additional mathematical structure. 
The theorem* proved here will be applied in later section* to sets with 
((eometricai or topological structure*- Wli™ riecratf ry fur truth R must 
be taken as finite. 

Out theory ileAl* with predicates detined on subset!) of a given base 
apace which we shall consistently denote by R. We use the following no- 
tations! conventions; 
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(i] Let R he an i*rhilrary set and & a family of subsets of H. UsLruj 
the letters X t Y r Z r ■ ■ - for subsets of R it is natural to associate with 5f 
a predicate QitiX} which it TKUE if and only if Jf e -^ 

{ii) We shall use the letters* and ^ to denote predicatra defined on the 
set of subsets of It. 

Wc shall uae the notation tf(X) sometime* to mean tin; predicate whose 
value for 4i given X is TRUE or FALSE, sometimes to mean a binary a*t 
function whose value ka 1 or 0. Whun we wish to employ the two senses 
in the same content we adapt the notation Pi^)~| for the binary function 
whys* value is 1 if f (JO is TRUE and if +IX) is l/ALHE. We will usually 
UM thiH only when there i& » possibility of ambiguity, e.g., to distinguish 
between r* < &1 - i, whkla is true, and 3 < ffi = 1"| F which i* false. 

(hi) Occasionally it will be convenient in raaimijles to una the traditional 
representation of <t>(X) as a function of ft "Kooiean variables" where 
fl - | if j. If the elements of R are t,, - ■,,*„, it La traditional Uj think of a 
subset X of R as an afc*i K Ti<ition of the value* 1 or Q to ^ ii«.:ordmK to whether 
the point J(, i» in Jt or not, i,*., ' V is used ambiguously to stand for the 
fth point in the given enumeration of R „ and for the aet function fj, £ Jt - 1 . 
This notation is particularly convenient when £ is represented in the 
form of a SLandard Boolean, function of two variables, Thus. X, V^ is a 
way of writing the set function 

+to - F* G X or Zj e X~\ . 

(iv) We need to express the idea that a function mav depend only on 
* whect of the points of R. We denote by Si*) the amallest subset S of 
iii with the property that, for any *ub**t Jf, 

*lX) -^nfl. 

We call £(+) the SUjyjeurf of tf. 

[v} Lei * be a set of binary set-functions on k, We say that t a a itrtfo-r 
/AresftoiW futu-.tion with respect to * if m ewh member * of * there corresponds 
a real number er, such that, for some real number d: 

*(jc>-ri>**w>r|- 

This is often written more briefly as 

We denote by £(+) the set of functions ^ expressible Ln this way. 

(vi> We now introduce the central concept of onfcr. The Q«r*J- of g:- is 
th<- smallest k for which there is a * satisfying 
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*£*^ \S(+)\ £* 

where IS(*>| i* th« cardinality of £?{#). 

Functions of order 1 appear Ln the literature under the name, of "linear 
threshold functions." H should hv noi*d that the order of a constant 
function is zero* hence the number r? in the definition of Il» can be re- 
placed by U (or any other number) without changing the definition of 
order. Nolc also that the definition is, unchanged if wt use "^ " "Sp" or 
"<" instead of ">" (assuming, when necessary that J? i& finite). 

(vii) * iii called a mrnk if there is a set A Mich that 

We denote thus function by # A . 

In point- function notation a mask is a function of the form: 

where j >, ;■ is thu subset /i of i?> In particular constant functions are masks. 
Lifieftr Representation. 
F£OFOs]Tion, dK JWfSfes ore iff vrder 1. 
PROOF. For each r£A define fc(X) us fjGA"|. Then 

In particular the functions f, $n<3 ^are of order 1. Similarly the functions 

iVy, *Ay, sO,y arc of order 1. But the "exclusive or," x &y h and its 
complement, x = y, are of order 2, 
Ekahple (i). iiVjsiVjei i» of order 1- 

[~ft + JCi+ft>fl~|. 

ft A ft A ft is- alao of order 1: 

r*i + «i + a*>2l. 

j|F 2 = fit + (1 - ft) > lT| ■ r*i - *a > Ol is of "der 1. 
ftVft = [~ft + t 1 - ft) >0~| - |~ft - ft> - 1~|, which is also jsc, >* s , 
re of order 1. 
Example (id), *, ■ ft, which is 

*,ftVftft- r*L*!-Ml- ft>U -ft) >07 
- |"2ftft - ft - ft > - l~| 
is of order 2. (Proof that it is not order 1 i* in |II.l 
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Kx ample (iiij. I,el jU be. an integer < JW < \R\^ Then the "counting 
him-Lbm" 

i H {X)=f\X\ -AT|, 

which recognizes, when X contains eiocrty ilf points, is dJ" order 2„ 
l^ftOOf, Consider the representation 

* M {X) - 1-(2« - 1) Z* + { - 8J ^xt fc Af H . 

■Ill I F-.l 

l\'ianyrLgureXtherewil|oe|X| terma r r with value i t and | X\ .{\X\ - l)/2 
1erms k,*; with value 1, Then the predicate ts equjU to 

**QO=n2M~lH*| -|X| -(|^-D + 1-AI ] >0"1 

■IK) the only (integer value of |X| for which this is true ]* \X\ - M- 
Note that the linear form far the counting function does not contain 

H explicitly. Hence it work* ins well for an infinite ftj> H ce R. Q.E.D, 
Example (iv). The functions rPH £ W~| md |~|Jf| * Wl are of order 

I becauae they are Hprewnud by rZ*. *M1 and T £>, ^ Af 1 ■ 
EXAMPLE (v>. We can obtain an arbitrary function /f|J£|} of the area 

<•: a hgure l>ym the predicates used in (iv) above by writing 

fix) _ fm ■+- £ (^(A) - /{k - i»- ri-K"! > *1 ■ 

The order of a function Cfln be determined by eiainining its representa- 
Uvn as- a linear threshold with respect to sets of maska. To prove this we 
Ml show 

[ jdt:oKKM i Positive normal i-uhm theorem} . Ecery * is a linear threshold 
junction with nf*p«J to fcbe seif o/ flit masks. 

PROOF. The welMtnown disjunctive- normal-form theorem for Boolean 
functions tells us tlm any Boolean function H^-^.xJ can he written 
in the fonn [DNF) 

<HX)= ViAX) 

where 

+iiX}=y; 1 *L 1 '--y{ h 

where for etch i and j', jv^ - x, or y rj = zj. 
We can writ* this, in linear form a&: 

becauEe, for any JY, at most Otic term of she DNP is nonzero. Hence wc can 
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replace logics] "V" by arithmetic "4 ." Furthermore, *\,n*x numerically 
*. - 1 — fj, each ik Dan be written in the form 

supposing that the negative terms axe at the right. Multiplying this out. 
we obtain un expression of the form 

where Z t is uf the form 

*'i " " ■ *w*il ■ ■ ■ *■■, with ( A Ll . ,.,*.. | C {<»*!, • ' ■.*.}. 
But fritffa Z / are ma&kt, w that ft ia a linear combination of mtukt. It 
follow* immediately that £,- EJ fc is itaelf a linear combination of maaka 

where each s,um integer and each J^ & mast. Q.E.D. 

Remark, The abnve construction shows not only that any Boolean 
function is "linear*' in the set of maaka in the '> - f Ea,^> lt~\ " sense, 
but is also linear in a stronger L >_^a.*." ^sr. lr. .« interesting r.hu« 
this form is unique, nnd is therefore entitled to he called a "norma] form." 
We call it a "positive noniuil form." To see the uniqueness,, suppose that 

and conaider the difference 

*-(^i^-lA2.)=Ik-ft]Z,-lT,4 

Now <>(X') must be identically wro, To see thia, consider first any «t 
X of one element i, , Then 

HQ tij,! = 0. Naze, consider any two-element X = \xtXj\; then 

ao all tw«-c]emeot T|*r_y|'» are zero. Similarly, by induction, all the >'s 
can be seen tn vanish. 
The proof of the positive- normal farm theorem implies n\&} the 

Theorem, if is of order k iff k is the amalksi number for which (hem exist 
a set * of itiaxkjf satisfying 

+ G*+\$i+}\ £k 
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EXAMPLE (vi). A "Boolean form" hm Harder no higher than the degree in 
its disjunctive norma] form. Thug 

X Q M* J i *J ** = X*»*i*i — X H v* ^ ^ r * 
so that the nejationa can he removed without raising order. This particular 
order-3 form appears later in a perception th;s1 rwugnizoa convei figures.. 
Uaingthis result we develop nome more eiamples of the us* nf the concept 
of order. 

THEOREM. If ^ bos order L and $ x has vrder 7 , Chen ^ ® f 2 and 
if, = (!.j. taw ftnjfer jS L + Og. 

Proof. The idea is io multiply together the: positive mask representa- 
tions ft ^i* - 8|> [Si* - ft) > OH to get a positive form of order £ O, \ Q*. 
i Um " > " foi - and " < " for ® ,) ThLa may not work in &ome c* ses where 
Iri — Si or Zj— flj. in such cawe* it is always possible to replace * L Arid 
fa by slightly different value*, al^ebraicalLy independent of the coefficients 
of T, and E : , so that the predicates are unchanged but exact equality 
never holds. 

AppJtciition. 

KxAMPLt (vil). Since f"{X\ ~ \~f\X\ % M~\ = \~\X\ * M~\~\> we 
conclude that tf" has order ;= 2, the result of Esample (iii). 

Question, What can be -said about the ordera of |~*i A^ s "l and p# : V +i] 7 
The answer to this question may be surprising, in view of the nimpLe- result 
of the previous theorem; it ia shown in |V that for any Didar n r there 
exists a pair of predicates i, and 1^2 both of order I for which {<y, A ^ 
and (in V^ ? ) have order > n. In fact suppose that R - A\JH\JC where 
A. W.AnrK'iiTc large diE-joint subsets of It. Then ^, — ^1X0 A\ > \XC\C\~] 
and ^ R — | | X f] B \ ~>\X f\ il\ | each have order 1 becau&e they UV 
represented hy 

ri^-i^>oi and rz *. - £ *. > <n 

■itJ =iec ^f=B ,,^_c 

but, a* shown id §V, ft, A ^) and (0, V *t> have high orders. 

II, Croup theory of linear inequalities. In thia .taction we con&ider linear 
thjtf&hold functions that art invariant under groups of permutations of 
the points of the oase-space H. The purpose of this h reahzed finally in 
SV, is to eatahlifth a connection Between the fleOrnetry of R Hnd the 
question of when a geometric predionfe can he a linear threshold function. 

Aft an introduction to the methods introduced in this section we Bisl 
consider a simple, almwt trivjjd eiample. Suppose w« wieh to prove that 
the function x x x 2 V -*j n t is not of order 1. To do ao we might try 1y deduce 



IW fclARYlN «UNSftlT AND SSVWOUJf HAFcRT 

a oontradietion from the- hyp^ltniEis that numkre *> & and * Can be found 
lor which 

'' i) *(* i , *0 - * t *t V F^ = Tax, -|- tfi, > # "| . 

We Could proceed direcUy by writing down the condition* on « and ff: 
ri = , iii - -* > #, 

*i — I? *e = V =3* cr a *, 

*j = 0, * t - 1 =S* £ ft 

In thi& simple case it is ea&y enough tu deduce the contradiction. 
But argument* of this Eort are herd to generality to mon oeDpha situa- 
tions involving many v;iriubLe&. On the other hand the following argument, 
though it may be considered more complicated in itself, leads to elegant 
KcneraLizatJons. First obaerve that the value of V^ ifl invariant under 
pernH:lEilJon of x, and * Sl that ia, 

Thua 

**: + fix, > ft 

a J 2 -+- fix, > ft 

yields 

hy adding the ineuualitieg. 
SimiJarly 

<**l • 0JC 3 S ft 

yields 

{(a + ^/2) * t 4- ({■■ + 0>/2) *, £ (. 
It folJows that if we write y for [q+fl)/3, then 

ifc, we CBn assume that the coefficient of jc l and *j in the linear repre- 
sentation of i are equal. It follows that 

tw= rijcj ><n or r*m-fl>oi 

(if we assume that the space X hag. only the two points jKj and x^h 
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Now consider three valuea of X, 

X(i= .\> |JC b l=Q, *|JC|-*£0, 

Xl-|*J. |Jf,| -1, y\X\ -i>€, 

X£=\x,.Xi}. \X t \-2 r T |Jt|-*£0. . 

Since JK U and X, -sotis/.y 0, and X T does not, the firU-drgrte polynomial 
l\X\ - § in j X\ would have to change dtmfterj tu;ie e, from positive to negative 
&nd back to po&itiue &s\X\ increase* from fei 2. Thra in, deafly impossible. 
Thut we ieam something abvul ^ by averaging it aver the permutations fkat 
leave it imnrianL The method i& sLmiJar tri that used in Haai measure 
theory. Iti f H ct h for order 1, it is the same method. 

The generalization of this procedure involves consideration -of groups of 
permutations on the set R and functions; £ invariant under these groups 
of permutations. In anticipation nf application t^ ftMhlAtrical problems, 
w* retail the mathematical viewpoint f?Qro which every interesting geo- 
metrical properly i& an invent of some natural transformation jgrmip. 

Let G be a group of permutation* of R v g £ £V and X C R< and define 

# J (Jf ) = **(*,>, 

Thua we define- an equivalence relation of +'s with respect to a Croup G. 

The Group I^varian^r Theorem. Lei 
(i) G be a finite group of permutations of It; 

i ii) * A? u set of predicates OH R closed under (;, i*- r *€* t jeO*#'£»; 
(ili) ■/■- i? tn £(*) and invariant under G. 
Then then* terVte a Jwwot repreaefttadwrt 0/ # r 

/nr Hift«^ the coefficients d t diywnd only on the ti~equiwteme class of *. ije., 

* ■ o #' ■* fit. - »<• - 
PHOOF. Divide + into equivalence classes hy the relation =c- 

Now let ^ - rZte+ l »*+(^) ^ °1 be any linear representation of * and 
ehOGM X such that ^(X> i,e,, X.*e*«»*!^0 > ° 
Sine* tF<Xj -if(X f ) 7 it follow* that for each £Gt7„ 
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Hinw the sum of positive quantities ts positive we can sum all such equation*: 

Z Z **#*cx) > n 

SLnue +— U?_l*j f the expression on the kft can he written: 

Z Z L*.#'-Z Z £*,+'. 

Hence, 



Z Z(Z^# J U)} >o. 



Now observe that the set 

because any j }ij*t permutes mernU;re of an equivalence class. Then also, 
Hence fra- uny £ 

So 

Z Z-f*^Z Z«vi# = T. (L*„-i)*- 

Since as * runs orer «, ** "covers" 4 £l „, then £, btiV , H , ha* the Emt 
value for all equivalent *\ i.e., Lf *£* it Lie^w-t depends only <,n ;. 
Therefore w* on denote ZrfcU* w -j fay p\ obtaining: 

i 

Z Eft*un>o 

where ££*) denotes "the equivalence class containing $," 

A similar a^umvnl shows- that if £>^{.-Y) <0, then ^ wi (X) 
<0. Thus f-rZ*4#>">l -rZ^^i#>0"|. W* shall moat often 
use this, thvorcm in the folly* ing form: 

Corolla R* 1. Any function *. of ordet k has a tiiw&- rvpresenWion 

#-rz**+><n 



UNfAHUr UNft£CQGNIZA&LE PATTERNS 187 

uMrt + it the m of masks of degrees £ k and a t — tr 4 uihemuer tffo) can be 
transformed into S{$') by an eiement of G- 

PROOF. The corollary follows immediately from the theorem and the 
otrwT^i-ivn that, for tnftx\c^ ? <f> A = ti 4 B if and only if A = B t for sum* & fc 0. 

tlnnoujiHT 2. Lei (■♦ilj — U** i* tAe rimwrnpflffirion o/ * into 
dvu ; la wiHw ffcissef frj.- (he relation -,■;,. Then if $ is in L{&} and 4- is closed 
under G r \f> can be written in the farm 

where NdX) = U#i*€*- ${X)\\, i.e., N { {X) is the number of 6 e of the 
i-ih typi* ttiUUXtkni under the group, that "fit" the argument X. 

Pboof. £ can he represented as 

-rz«. l*>°i -rz«<iw>><>"i. 

CurOllaht 3. {'Van TmiviAUTif up- Invaxiakt PREniLAiisa ur OarEK 1). 
Z#J G it' any transitive grvup of permutations on R (transitive means.- for 
euery pair p k i? £ R there is ag^G such that pg — q). Then the ntity firsi-order 
predicates invariant under G are of the forwis: 

*cx)-rixi >»i> 

or 

^(Jf) = f" | X | < ia"| , for some m. 

Proof. Since the group is transitive all the one-point predicates c-| f | 
are equivalent. Thus we can assume thftt 

if(X) — rZfe« a +i r) >*1 (or with same other inequality sign) 

i.e., the coefficient & Ls independent of p. Bui X.j, e j( ir* i p | > can he 
transformed into Z?f **IM ^ *■'" ^ nr ffl -* ^ ^ >lr ^^ a similar [argument 
proves the Corruspanding aflaerlion). But Zpf**;H~ 1^1- Tliiia order-! 
invariant predicated can A<* rt4il.Iii.rijf mure 1hiin define a count on the 

cardinality or "area" of figures. In fact, hr. unh-r I predicate is a mcasun 1 , 
3ml 1hc order- 1 invariant predicate is the Haar measure. 

III. Applications ol die jmmp itivpriAncc ibrurtm, 

TAt 1 Portr>' Futoftfn... In this section we develop in some detail the analysis 
of the particular predicate ^W defined by 
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Our interest in -fa^n is threefold: it its interesting in it*df; it will be used 
for the analysis of other more important function*; and, Especially, it 
LUu&traLes our mathematical methods and the kind of question they enable 
us to diacusA. 

Theokjsh. ^ PaH is q/ order \R\. 

That is, to compute ^p^H. requires at leant one predicate who** support 
covers 1 he a;hvU space R ! 

P*«*. Im G be the gioup of all permutations of ff. Clearly ^ FAR is 
invariant under G. 

Now suppose that ^ = fX«^*. > <*! where the *j are masks *ilh 
|S(rf,) | 5 K asid thcu, depend only on the equivalence classes defined by -ql 
Since masks with the same support are identical, 

*, = „ *., ^ | S( fc ) | - | Sfo,j | . 
Thus 

where *, is the set of masks whoa* supports contain eiaetly j elements. 
We now calculate for an arbitrary subset X of J?, 

Since *(X} n 1 if S£#} C* md otherwi*e, Q(Jf) is the number or sub- 
set* of X with j elements, i.e., 



C t iXi= ('*') 



which k a polynomial of degree j in |X[„ 
It follows that 

ifl a polynomial of degree X in |A|, *ay P({X\) r 
Now consider a sequence 

of J J! | + i nested subsets of fl, and the -wquence of values 

P(JJf,,|)-O f P{jX,\i=l. P{[X s \)=Q,... t PI\X i/!l \). 

This implies that P< | Jf | ) changes direction | R\ times as | X| increases from 
to | if | . But since P is a polynomial of degree ff, it follow? that K - \R\ 
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From this urc obtain the 

Theorem. If fpAR-E £(+) arid [/ * ctHitaitts t>t%ly masks, then * contains 

till the ttia.-yh.-, 

PROOF. Suppose, if possible, that fam £&(*}. that + contains only 
llnn&ltE, and the masJt whos* jsupfwrl i& A does, tint belong to +, 

Ut *P«R-rZ*e#*-*> l. Define, for any *, *-"<X) - ${Xn A) 
Clearly ^ftm,. the parity function for subseta of j4 f is of order \A\ by the 
previous theorem, 

Now rounder #* for *€ *. If S|*) C A, clearly ^ - gk If £(4) it not 
a E-ubeet of ,4, p.* is identically aero since 

It followE- that either S{tf> A ) i& a proper subset of j* or # x is identically 
wro, Let £ 4 h« the set of masks in + whose supports are subsets of A. 
Then*™i= |~X*e*Aa«tf > Q~3- But for a]] 4<=*\ \S{+)\ <|A|. It would 
follow that the order of ^j!^ is less than \A\ r which i* a contradiction^ 
Thus the hypotheses are impossible audi the theorem follows. Q.E.D. 

COROLLARlf 1. J/ £fah ££.(*), ***« + fOtsi contain at t?a$t one 4 for 
which 

\sitf\-\k\. 

The following theorem, also immediate from the above is of interest to 

students <?f threshold logic; 

COKOIXftMT 2. Let + be the Mi of alt ^tap. for proper su*#efs A of R. Then 

The following theorem gives a hint that rerUiin functions, that might 
be recognizable, in principle, by a very large perception., might not actually 
In: realjzuble in practice hecansfi uf huge coefficienta. 

Coefficients of the Parity .Function. Suppose that we hnve a fH^tf, > 0~| 
thai rcw)gniw& Parity i\X\i with. masi.*, htf, u& suppose that the reeoitnitiuti 
is reliable, *,&„ that 2£n,*f> 2 for odd purity, and ^u,*,-<0 for even 
parity. If we apply the full permutation group, we ebtain the same reliable 
discrimination with a set of "average coefficients" a^ aU equal for p-'s of 
the same order. Then we obtain the inequaJitieg 

«■ > 2 > 

J H f \ 

i» 2 + "■*■<: < ]■ or 2J ( ■ ) <': > 2, if n is odd, 
a? + 3or E + 3a t > 2 \ < 0, if n is even. 
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Subtracting swcc*B&ive inequjiLLtn's, define 

—+?[ ev)-m — + $g:i)- 

so that for a]] i», 

(-l} n D (1 >2 Of [f- l)-£>„- 2] >0. 

Using- theo*. inequalities, we will oblain a bound un the coefficient* (ct, J. 
W« wUJ sum th* inequalities with certain positive weighty ehootw any 
JW > 0, and conKUter 

f(f) \i-im -a]> 0l 



Thvn 



The left-hand «tdc Ls 

■r 



s&-»^(i)(r)-£&-»'«-(i)(r) 



•II 



= ftil + l<-l) J 

&(■ w? have the 

Theorem. Far «eA M r 



LINEARLV UNEECOG-NIZABIE PATTERNS 14] 

These value* iudd f«>r Hit? average, ao if the coefficients of each type are 
not equal, some mast be even larger! This shows that it is impractical 
to use mask-like d's to recognize purity-like functions: even if one could 
afford the huge numlw of pX one would hjve also to cope with huae 
Timjjus <>f their coefficient^ 

Kemjikh. This haE a practically fatal effect on the corresponding learning 
machines. At lens! 2 |ft| itustssice* of juet the maximal pattern h required 
if"ir;ir::" !!.■.■ uui-j cue iiuiiVnt ; :i;tii:illy ::.( r ,1 ijjii i ::n - _ li- far wots* heca\:*e 
<j( the unfavorable interactions with lower order coefficient*. It follows, 
moreover that the information capacity necessary to stun; the Bet | f , j 
of coefficients is greater than thai, needed tni store the entire &et of patterns, 
recognised hy if PAP — that is. the even subsets of R. For, any uniform 
representation of the a.\ must allow \R\ bits for each, and sinee there 
are 2 |fl| coefficients the total number vf l>iUs required is |ii| ■ 2 |fll . On the 
othci hand there are 2 1 *'" 3 even sublets of R, each representable hy an 
|J!|-bit sequence, so that \R\ -2 |J "" 1 bits wouLd suffice to represent the 
subae ta- 
lc .should also be noted that V- iv.il is not very exceptional in this regard 
because the positive normal form theorem tells us that all po^ible 2 1 ' 
Boolean functions <:a« he *i> encoded as linear threshold functions in the 
set of All anasks. Then, on the average, specification of the coefficients of 
each requires 2 S] hits.. 

Another predicate of great mterest is associated »ith the geometri* 
property of "connectedness: : ' Hs uppUcatioi* and interji relation is, deferred 
to $Vt the basic theorem is. proved now. 
The "Ont-in-a-bvx" T^kpt/h. 

Throrkm. JjeX A lr "' r A m be disjoins subsets of R and define, ihe predicate 

!-£_„ cfceTie is at Jeust one point &f A i.i cacVi A_. Then :"/ /ur oii i., | -4;| -* im s , 
the order of j- is £ m. 

CcmotLARY. J/ £•= AjUAiU '"U^*f ^ on k f of $ i* at least the 
order of (jfi|/4)^ 

Proof, For each i»l, ■■■,» let G\ be the group of permutations, of 
fl which permutes the elements of A, but do not affect the elements of the 
complement of A,. Let- G be the group sicnerated by &1J uJurrnmls ai the 
Cv Clearly £ is invariant with respect to G. Let * be the set of masks of 
degree K or less, To determine the equivalence class of any t £E + consider 
the ordered set uf occupancy numbers 
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Then *, = (j 4, If, fur each i, |£(#J n -4,1 - |$ffe) H A,| . Lei + u +, h - . ■ , * w 
be the equivalence clawes. 

Now consider an arbitrary sec A' and an equlvatenee class *,. Wcj w L sti 
US calculate the number JV,-(X) of members of *, .Ratified by X, i.e., 

tf/JC) = H#|#E*,AS{#)CJf}|. 
A aimp]e combinatorial anenment nhcvws that 

|xnA,i \ / \xr\As\ 



where 



fy\ _ j(j- i) ■-- ty = it+ i) 

W ji! 



and 4 in an arbitrary member gf *; P Since Lhe numbers | SM n Mj| depend 
on]y on. lhe classes * ; and add up to not more than K, \l follows that -^(X) 
can he written as a polymimial of decree K tnr Jess in Lhe numbers s, - | A' n .4,| 

Now 1*1. 4, - | £*,* > G~| be a reorvsentation of f at a linear threshold 
function in the- set of masks, of degree Less than or equal to X. By the 
argument which we have already used several times we can assume that 
a 4 depends only on the equivalent duss of $ and write 

E«MX) - £ft L #{^) = ffti^Wf) 

/-J *€+J ^-L 

H 

which, as « sum of polynomials of degree at moat K, in. Itself such a poly, 
nomiai. Thus we can conclude that there exfels a polynomial of degree 
at most K, 

with the property that 

MX}- pQix l ,—,x m )>Q with ^-|XnA:j"| 

i.e., that for all i, ^ jfj £ 4m : 

^.'■■iO>c# (Vi)fa>0). 

In Q$x t , - ■ -, jt„> make the formal substitution, 

*-{<-<2i-i))*. 
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Then ^(j Lp . . ...jf„> becomes a polynomial of degree at moat 2 K in t, Now 
let t take on the vAlues f — r 1, ■ ■ -, 2m. By property [<f>) Q must be positive 
for even. ( and negative or j-.cro for odd I. By counting the: number of chants 
of sign U is clear that 2K £ 2fh i.e., K ^ ra. This completes the proof. 

JV. The and /or (heurcBi. We have already remarked that, if H = A\JB 
U C the predicate 

^(Jf>- Tl^n^j >tXnC|1 i*of order 1. 

And stated without proof that 

is not of bounded order as \R\ become* large. We ahall now prove this 
assertion. We can assume without any loss of generality that \A\ = \B\ 
= \C\ And our formal statement is that if irfX) is the predicate of the 
stated form for | ft{ - iik, then the yrder of **— = as *— <* . The- proof is 
similar to that used for the parity theorem. We shall assume that the order 
of itr) is- bounded by JV for Bill k and derive the contradiction by shewing 
that the associated polynomial* would have to satisfy inconsistent condi- 
tions. The first step is to set op the associated polynomials for a fixed k. 
Wi do this by choosing the group which permutes within the Sets A* B. 
('.. The i-qui valence classes of masks are then characterized by three numbers, 
Le. r \Af-\ £(4) |, | Bn S(*) | and j C O S(*J | . The number N^X) of masks 
in thus equivalence class satisfied hy a given act X is 

N(X)-( lAnXl \ x ( lBnX] \ v f |CnX| ^ 

\\Ar\8M\/ ' \iBn$to)\) x \\cr\sw\) ' 

If |£faj, i jV this is clearly a polynomial of degree at most jV in the 
three numbers 

x-\AC\X\, y-[BnX\, x-\cnx\, 

The group in variance theorem say* that if 

**-rZT*+->0"| 

when + is the set of masks with |S(*>| Si N, then 

where i runs over the set of equivalence classes of <p. But ^ajA^X) is 
a polynomial of degree at most jV in *, > and z. CaLl it Fifr.?,*}, 

Now, by definition, for possible values of i,y, * (i.e., noncurative integers 
£ k), / J ^JC,y..ij > U if and ruslj if j > z and >■> r , We ahatl show, through 
a aeries of lemmas, that this, cannot be true for jh.II k. The technical details 
of these lemmas are not essential for thr subsequent sections. 
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Lemma 1, jM Ps[x t y t et be an infinite deguciuc of polynomials of fistd 
Qtyrt* *■ toitk ft* property iitat for all positive integers x r y, % &$* than k, 

(A) 

ii * «r y £ t ** P t {x m y,zi so. 

Then there exigU y riunzero polynomial Pix.y.s) of the ssuhj decree u with 
the property that the implications (A) hold for all positive integral values 
of j. y, z. This follows from the foJlMvinj, Compactness argument: Write 

Pii^y.i) - 2 C,., m ; {i,y, *) 
i-] 

where Bid,)', i) is no enumeration of the monomial* in vari&bles ij y 

arid z. We can assume J^^b^ 1 since- the hypotheses remain true if P k i& 

divided by J^C^, Now the handed sequence C Ll , C ri , - --, C,,], " ■ J 

must Contain an infinite convergent su hseqiieiKe 3 3 of the integers tor whicn 

] CVJ A £ Sj j converges to a limit. Bay C 5 . 

Now consider | C'a.i I * C Si j. There must bt- ;tn infinite subsequence of 
Si t say Sj, -on whkh this converges to a limit, say Cj. Continuing in this 
way wfr find & subsequencH £™ £, of the integer* And a set of numbers 
C,---C r Buch that fCV|*ES] converges to C. for all ifir. But then, 
for k£S, P^3t t y t $ converge* to the- polynomial 

r 

J^Ijt. > . a) = X ft m, (jc h ^, f > for all t. y, z. 

J-L 

To *ee that P{x,y,z) has the required properties, choose any positive 
integers *&, y^ *u, For values of * smaller than the largest of these number*, 
noEhing can he said, about F*<s ft y*,^- Hut fur all sufficiently large k, 
Pti**yay*& must he nonnegaliw if **>!<, and y i) >s il (and nonpositive 
if *<, < z or y.j < a„J. It follows immediately that PU^y^z^ is nonnegative 
(nonpositive-J under the same conditions To see that P i* nyoMfro note 
that £>*=!, 

LnM.hU 2. If a polynomial ffa.B) Wtitfitt fa feUowing conditions far all 
integral uataen of a and 0, then it is identically zero: 

(B) a>uandfl>0=£- /(«,j9) £ 0, 

{CJ * g «r $ < -*■ /{a, j3) £ 0. 

Pk<j<jj . Suppose that a polynomial of degree N t f (*>$}, satisnas the 
conditions (B) and (C) and is not identi-cally zero. Without le**¥ of generality 

we can suppose that 

ftjMifl - a s gm + r( ff ,tf) 
where gtri) is not identically zero and r{ a ,<j) has degree Jeaa than rV ifl a. 
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For any .0 for which gifl) t* 0, there is an n r , :=■ G such that 

|a*4(A|>JrbiA.A|. 

Thus /{p(i,fS> ha» the same sign as ■aj'stftk, Le., a» fijs) since «f w positive. 
It follows from <B) and fCj that 

ID) 

(The conditions (D) hold for Alt 0: if aiff) H & by preceding arguKHinLi 
Lf£(£) = 0, ta-utoLoflously,] We now derive a eorjl-radic Lion by considering 
separately two ess**: 

i,&\ jV eiwn, Since £.<£> i& not identically aero, there is some fl] > for 
which elft)*u\ By (D) p £(ft) > 0. Thua a"Vft) > «* that for \*\ 
RuffidentJy Large 

a*tf(ft)+r(a>ft) >ft 

i.e., /(u,S(|) > 0. But we are free to choose a negative value of a r i.e., we 
can find h*, ft such thai 

ceo <. and A*u,^:> > 

which contradicte- (Ch 

(b) N odd. Choose ft < for which tf(ft> # 0; then g(ft] < 0, by (D). 
Choose negative a D as before. Then avgi&d > and /(an, ft) > 0, again 
contradicting (C). Q.E.D. 

Lu&iaiA 3. JVo Fiensen? paJ> , /zanz*Gc' P(i,y,s} eon satisfy the foltf/wing 
conditions for ail positive integral mluet of x. y, z: 

x > * and y > x -* P{x, y, z\ £ 0, 

i ±2 or y Zz-± t*ix,y,t) |0. 

FftOOT, Suppose- th*L P(*,y,?Ji has these properties. Define QiccP.e) 
— F {i ■+- a , z -r f) p z>. Let Af be the highe&t power of 2 in Q so that 

where /i i& of degree Leas than Af in r. 
Now choose any a,-, arid ft for which /(a. ,ft|> ^0. For suMidently large 

(a) $, 4- a* > and 4d + ft > 0, 

It follows that 

ft*:, ft) £ ** Q(a , flp, z,) £ 0, 
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Thus 

«,j > and J? n > -*> £» + ir > S« F and ^ + ji* > 4, 

and similarly n c <0 or ft < =£»/(&£,, ft,) £ 0, But this ks true for all 
»cJ3 t . ThuB by the previous lemma., fia,p) = G. It follows that P(* f y f i) 
is of degree siero in i, whieh is only pessible if it is identically aero. Q,E.D, 
This concludes the proof of the AND -OR theorem. It ta clear that the 
reason the theorem, is- true has to flu with the algebraic geometry of the 
"occupancy" pelynomiaLs. If it were tint for the constraints concerning 
integer values of the variables* the theorem would he an immediate coft- 
sequence of Bezant's theorem, 

V, The '"order- ILmilcd" pcrccplmn. 

The Order 0/ .Sown* CinvrruKn'cui r^vdicaies, Now we consider the problem 
of computing the order of A number of Ln teresting geometrical predieatea 
Aa a first step, we have to provide the underlying space R with the topologi- 
cal and metric properties necessary for defining jfOOrnvlricn] figures;, this 
was not owcsMssiry in the case of predieatea like Parity and 1 illicit related 
to counting, fyr ihcuc arc not really geometric in character. 

The simplest procedure thai is rigorous enough yet not too mathe- 
matically fussy seems to be to divide the. E»if:LiuV-i' n o'ane, E*, into squares- 
as an infinite chess board, The set R La then taken as Cta .«( 1?/ .sgtmrrf.Y, A 
figure X \fi rj ? is then identified with that set of elements nf R — i.e., that 
collection of aquarea— that contain a1 least one point of X, Thus to any 
subset X of E 3 corresponds the subaet X of R oVJincd by 

Now, although A" and X are luijicudly distinct no Eenous confusion can 
arise if we identify them, and we shall do no from now nis. Thus »t refer 
to certain subset* of fi as "circles, 1 " "triangles." etc., meaning that they 
can be ohtained from real circles and triangles by the map X^X. Of 
course, this means that near the "limits of resolution" one begins to obtain 
apparent errors cf classification because of the finite "mesh" of R. Thus 
* Miiiali circle 
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wiU not look very round. 
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Whun it is nucussary to distinguish between F and t' we will say that 
two figure X, X r of E L are in the- same fl- tolerance class if X = X', In 
this we follow the general mathematical ;jj'j]jjihh:1i proposed by E. C 2eeroan 
for treating this kind of problem. To avoid inessential questions of how 
the group- in variance theorem applies to infinite groups* assume- bdow 
wkn^n necessary that ft has the lorwdsl topology 

We begin by listing some geometric predicates of rather small order. 

(a) 4=1. ffboi we soy "geomctnc property" we mean something that 
is at least invariant under translation, usually also invariant under rotation, 
and often invariant under dilatation. The tir$1 Lwu in variances combine 
to define the "-congruence" group of transformations and alt three the 
"similarity" group, For k — l h just the translation group Suffices for the 
druup IrwAriance Theorem to tell us that alt coefficients are equal, hence 
the only patterns that can be of order I are those defined by a single cut 
in the cardinality or area of the set: 

t=r\x\>A~\ or i?-njri <j*i. 

Note: 11 translation InvtrUnce is not required, then order- 1 can compute 
other properties, i.e., concerning moments ahnut partictilar pi.itn.ts yr m;$. 
However thes* *re not "geometric" 

(b} k — 2. For k — 2 things are more complicated. Aft shown in $1 it 
is possible to make a double cut in the area of the set, hence we can do the 
counting trick, And rHCognrae thtKw fijiuns whose areas are 

*=rA L <|F| <A~\. 

Un fact, in general we can always find a function of order k that recognises 
the sets satisfied by any k inequalities concerning their cardinality.} Now 
consider only the group of translations and masks of order 2. Then two 
masks jc 3 jc £ and aJj^ aTe equivalent If and only if the difference t*teft)rj 



x, - x* 



and x[ — x'} 



■rt equal. Then, with respect to the- translation group, « figure us completely 
characterized (up to k — 2) by its "difference-vector spectrum," defined 
as the sequence of the number* of pairs- of points separated by each possible 
directed distance. The two finm; 
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have thv same difference-vector spectra hence no order-Li predicate can 
make a cJassLfication which is both translation invariant and separates; 
these two figures. Similarly, 




ind Y 



YJSSC//^ 



ei 



■.H* mdistinguisnabJe. while 
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m 



i- 



^ 



- d F^f 



^ 



« 



have different difference- vr*lv>i" spectra. 

If we udd the requirement of in v* nunc* under rotation, the tatt pAir 
above- becomes indiatinguiEnablc, fur the spectra now relevant classify" 
together all differences of the same length, whatever their iirientation.^ 
An LntereEling pair of fifljna'es rocationaJly dlsC-mef, bin. si ill indistinguishable. 
for k *- 2, i» th* pair 
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which have the same direction independent diElanec-- between point-n^ir 
statistic*. There is an interesting theoretical direction here, but we will 
not atop to look Into It. Many interesting proposals for pattern rceogni tion 
mi>di i nts 4iri' r^Uted ff> the theory of these geometric spectra. The classic 
paper o£ Bledsoe and Browning \1\ is related to thi&, as Ls the work on 
"integral geometry" of Novjkoff i 4 J . 

(c) M - J. As i intrejwes h the dass. of reflliraftble- distfrinilnationB- growa; 



J Nnte Lhat we did na( nUnw Kflertiojit-. yet lli¥b* i^5«ttnrulLy ffpfmritc lisu>r«i> arft now 
cunfaKd! One ahoidd be- cnutwui nbiwC u»ii£ ■"iriiurtion" bcTE. Thw (hwiry of mtationa] 
invSJiariM rH^'fra (■■SKl'llI ellaiitloffi to IK* efftrt af Lht diat-rele r*tjneJ AppraiLmBCrtP, 
bin cur pruuimfltily bf. made ainuaical by ■appGkntiiin uf Zesm&ri's rucLhrfuU; :dt Lh* dklAli- 
I kin "croup, '" Lhcr-r arc Krkira diliiniiliet 
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nn<i our detailed understanding wanes. It is interesting to discover that 
the predicate 

1>iX) = \~X bjAsinfckn solid > convex figure"! 
is of order £ 3, as noted in the I nt nod uetiort, because 
<i>CQ»vEs.i^i) — TX r oi G^ a^d i£X and midpoint Ca, tij ^ X~\ < 1~| 

is of order 3. Presumably this predicate tunnot be realized with order S. 
It is not dti1tcu.Lt to show that the set of solid re*;1 <i ngle-& {with axis parallel 
to the mesh of if) can be recognized by a predicate of order 3. This is true 
also for the set of hollow rectangles (with borders one square thick), It 
is much more difficult to snow, but true, that the set of hollow squares has 
order IhrMi" [riLui Liwly wu 1 might suppose that at least order 4 is required 
to insure equality of side length*.. 

Another example of & predicate that can he realized with fe— 3, for 
any jn, is 

|~the points of X are collinear, and broken into 
not more than a segments"! - 

{d) it — 4. Using the fact that any three points determine 1 £ circle, wc 
can maltea perceptrem with fflHSki of order k — A for the following predicates: 

${X) — (~ X is the perimeter of a complete circle"! ■ 

Proof.* Define, for all concyehc quadruples of points in R\ a, b, c r d 

#*g(X> - p»€* w^ b£X tnd c€X and d$X~\ 

And then realize ^ as 

T.t.T.d 

Many other curious and interesting predicates can be shown by similar 
HrKiLmeriu Ln have small on I it.-,. On*,! should be careful not to Conclude 
that this means that there are practical consequences of this, unless one 
is prepared to face the fact that 

{&} large numbers of .p'e are required, of the order of f?*" 3 for the examples 
given above. 

(b) the threshold conditions aie sharp, so that engineering considerations 
may cause d i fikullics in realizing the linear summation, especially if there 
is iiny problem of noi*e- Even wilh sirnjils si| u a re ■ iikjL »uise, for i — 3 
or larger, the noise grows faster than the retinal size. 



An alternative mdhirii i:. lit :ir.P£rj'.i- Lht curvitiiru nt" l:n* elements. Ttiii lend* Irc inter- 
intuit quratttins atuwt Ih* pweWwi o! (lot)* I UmclKnw Lhei f*» be ipprtoninemJ Ijy suin- 
iDitidn of Idcb! «J«THfilE with ■ given jmciaioai. Curvature require* ord*r t, in a raise. The 
predicate dehintsl h*re admil4 v \'if uiinilvraUjii; cxceplic-nts. 
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fc) a very slight change in the pattern -definition destroy* the k«w- 
ntzabUity, 

Furthermore, in most ease& the™ will be more efficient machines, for the 
same amount of hardware, to realize three rather simpJy- defined partema. 
Low-order recognition has often the character of a "trick." and one cannot 
generalize freely. The AND -OR order theorem belli u* that aome Hjrap fe 
rebtitmB between SLmpJ* oropertiea of fignrea can be prohibitively hard 
to reccifinise. 

VI. CouJicrtirity: A geometric property with unbounded miter. We define 
fiOwwctedpimk aa follower 

Two point* of R are mfrcimt if thfly aie squares (in the map J'—F) 
with a common edge. A figure is connected if, given any two points P lr P t 

of Uie figure, we can find a path through adjacent square* from P, to P t , 
Theoken. TAe pmHt&ie 

ifriX) - \~X is connected ~\ 

has arbiimrity large orders as \R\ grvws in site. 

Pfcoor, Suppese thit Qf) could have order < m. Consider an array of 
(2nt + l) X<im J adjacent squares of R arranged in 2m + 1 row* of dm 3 
square* each. Let (^ be the &et of pointa shaded in the diagram below; 




row fl: 



i.e., the array points whose row indlcea Are odd, and Let L bjc the remaining 
aquarea of the array. Let 3 1 be the family of figures obtained from the 
figure Cf, by adding auhsets of G 2J It ia clear that if F £ -^ it ia of the form 
Ir / V , *h>re F : .J i,'.. N- ::•■>■ /"will hu i'l-iuiei M-ii iJanc. only if it., h\ flin- 
tain* at leaat one square from each even row- that ia, if the set W t satiates the 
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X:i 



ib one-in-a-box" condition (see end of ^3) P The theorem then follows from 
the One-in-a-Box Theorem. 

Toeeethedetajlscfhow the Ona-in-si Hern Theorem is applied, if it in not 
already clear, consider the figures of family 3" a* h subset of all pos&ibic 
fifto^? on fi. Clearly, if we- had an order-£ predict* that could recognize 
connectivity on R, we could have cme that worked on >; namely the same 
predicate with constant zero inputs, lm \\\\ variables not in (he small array. 
And since all points of the odd rows have always value 1 for rlfcurc* in ■&, 
this in turn means- that wt could have an orders predicate to decide the 
one-in-a-boi property on set G,i namely the same predicate further re- 
stricted 10 havirte; constant one inputa to the points- iti G u . Thus each Boolean 
function of the original predicate is replaced by the function obtained by 
fining some of its variables to 3eh* and one; this operation t^n never increase 
the order of a function. But since this hist predioate cannot enist, neither 
n/iin the drst. 

An Exampte. Consider the special case for k - 2, and the equivalent 
one-in-a-hox problem for a G r space of the form 



2k 



&Z, 






h>\. 




mmm 



in which m - 3 and there are juat 4 squares in each row. N V w consider 
&4> of degree 2; mtwil show that it cannot tharac-ten™ I he connectwlnesE 
of pictures of this kind. Suppose thai * - |""Z*i*.> *1 *nd consider the 
equivalent form, symmetric under the full group of permutations that 
interchange the rows and" permute- within row*,* Then there are just three 
equivalence-elasses of masks of degree £ % namely; 

single points: *,' = * ih 

point- pairs: ^ J = * iX) fo and *, in same- row) r 

puinl-paira: #" - T;Xj| (^ and *j in different tow*); 



N-ule thiii: Lhk » nut Ll^r 



KBmc g*;,^,, used jn ^Kjvin^ tr,*. jjEmtMl ih«tp*,n. 
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hence any order-2 predicate must have the Jorm. 

where jV, N" r and N li are the number of point aeta of the respective ti 1 p*s 
in. the figure X, 

Now consider the- two figures; 




^2 



1 



x : 











mm 














m 





























In each case one counta: 



JV 1 



JV" - o, W B - 9; 



hence she form \i) tiaa the aame value fur but!) figure. But X-> is connected 
while X^ is notf Note that here m - 3 so that we obtain a contradiction 
with | j4 ; j = A, while the genera] proof inquired |Aj| -4m 2 — SS. It is 
known also that if k — G.. we can get a similar result with | AJ — IS. 

The eaa* of * = 2, J» — 3, | Aj| — 3 is of order 2, since one can in fact 
express the connecLivity predlcaLe for that space tts 

*- fN L (Jf ) + tf H <Jf ) - 2iV"(X) > 41- 

f.YiMwisi! Oiflftrtliwliy, It shouJd be unserved that the proof of the previous 
theorem applies only to a property of connectivity in its claaaicaL seme 
but to the stronger predicate defined by: 

A figure X is "cutwise disconnected" if there ia a straight line L ouch that: 

F dots mtt intersect L and does wt lit mtirtiyto one side of L- 

The general vonnt'clivity definition would have "curve" foi L instead of 
"straight line," and nne would expect that thia would require a higher 
order for its realization. 
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tteiatww Btlisieen Pttxxptrons. The study of the order of predicate* is 
often facilitated by the redaction of a tiiwn pneiiiune to another simpler 
one. Although we do not have a satisfactory theory of any class r>f reduc- 
tions, or even a dear enoug h insight into the nature of the- relations which 
might play a. role antlogfliu t<> ''liPTiiCHmorphism,' 1 "quotient" and so on 
in more developed areas- oi' mathematics, the following exampLea arte, wt:iu\ 
in f;;." irnhi- \if:\'.\\r<i\i'--n*imt: iri:ii::.ii-. ;■-. interesting aiea for future research. 

(a) Let us. say that a pereeptron system, P, i» defined by the basic set 
J? and a set -T of predicates on subsets of R. A second pences>tron system, 
P\ is « subperccptron system of P Lf the basic set J?' is a subset of R\ and 
ifita s*t of predicates *' i» that obtained by relalivisine; the members of 
+ to R\ i.e.., aU predicates *"' £ *■' satisfy 

X£K'-** f {X) = *{X) for some #€ + 

<ind Jill predicates 4>' satisfying this condition are in * f . Clearly the order 
of any predicate of the form f' for F' is at most that of f for P. 

ib) Isomorphism must he Riven the following natural sense; Let P be 
defined by R and * and F" by it' and *'. Then an isomorphism, /, is an 
isomorphic mep /: R-*R' of the sets R with tie property that for each 
#£=# there is exactly one $'fc+ satisfying #(*) - <t>' (fix)) (where /(je) 
-\pGR'\~iqEii:fia}-p\). 

(c) P i» obtained from P by a COHap^ing epvnition f, if / is, a map from 
points of ft"" to disjoint sets of R r i.e., 

P i^ 9 => f<jp) fiflo) - I. 

A predicate ■£' on jf is, ohtahied from a predicate ^ on A bv the collapsing 
map / if ft*') - *{frX')J, for jt 1 C «'. 

Theorem ( t]omLFS[Mti Theo-hism). if / is a collapsing itwp /nun J? to 
ff" and ^'' is eAlajntfd /rom a predicate ^ fry f F (tew ute ardtr of £" is jw)( ^rrafcr 

Peuhjf. Let ^ = r S* CT ** ^ "H where # ia the set of maska of degree 
less than k on S. Now for any Jf' C^'. 

i 

We neit observe that (1) remains true if # is replaced by the set * of 
maalta 4 for which s(#) Cf(R') t for if 

*{*}<LfU?h *iflX*)j -a for *ll Jt* c *'- 
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Now for *£* we iiave 



in uit: 
Thus, 



■(♦JCUlApHpG/f). 

^(#)cUi/(p)l/Wnf(#)^A[, 



■*/a')3U| /(p) , ftp) n*("+) * a J 3 rM 

i.e., JP D (p| /(p) ni(*}p«ij# /(A"') D *(#) -*■ +(/(X'j). On the other 
hand, if ? (/(X')>, i.e., /(A- 1 ) ^i(#], it follows- that 

■nn/Cjfln/fel - Aforp *o. Thus #</<*')} -|~X'C (plftp) nsW^|T 

In other words #(/{*')) ia a mask on ft' with support 

jpj/(p)rv(+) p-aK 

But Bin.ce the $et* y| the form/(p) are disjoint, for different p , it follows thai 

J1p|Ap>ruW »*A}| £ |«(«)| <*, 

Going back 1y Equation (1.) we em, then, that ^' ls represented as a linear 
function of masks of degree lens than *. Q.E.D. 

Hii/jTMtn's finstHrt^n / w *„, We shall illustrate the application of 
the preceding concept by giving an alternative proof thai ^ w has no finite 
order, based on a construction suggested to ue by D. Huffman. 

The intuitive idea ia to construct a switching network which wOl be 
connected if an even number of its ft ewitcbes are in the "on" position. 
Thus the connwicdneas problem 'a reduced to the parity problem. The 
network Lb shown in the diagram for ft - 3, 



- *j — *i jc, . 

■J", JTs— JTj- 

-* — ^ —*t~ 



" x l *s Jfj" 



The interpretation of the symbols x- and i, ia as follows: when r,- j s in the 
"on" position contact is made whenever x, appears., arid broken whenever 
k, appear*- when x, ia in the "off M position contact is made where jc, appears 
and brokeu where x L appears. It is easy to see that the whole net is con- 
nected in the electrical and topological sense if the number of switcbeH 
hi the iL on" position is Q or 2. The generalization to n i* obvioua: 
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fa) List the terms, in the classical normal form for ^ PaK considered a* 
4 point function, which in the case fl vxn can b* written: 



^PftR^l 



*i*i 



V 



^ V *j*i*a Ki ■ " I, V " ■ V J£]4C S ■ ■ ■ Jf„ 

(h) Translate this Boolean expression into a switching net by interpreting 
conjunction a* swricK coupling and disjunction aa parallel coupling. 

(c) Construct a perceptroti which "looka at" the position of the switches. 

The- reductive argument, in intuitive form, is as follows; the Huff man 
.switching net can be regarded as defining b class. J* of tfeomeiric figure* 
which are connected cur noi. depending on the- F>Ari.ty of a certain g^f, the 
set switches in "on" position. We- thus sec how a perception for f^ on mie 
set, ft can be used UB a perceptron for ^ yfjl on a second *ct R', As a p*r- 
ceptroii for ^ru,, it must be of order at least | Jf | . Thus the order of f^ 
must he of order | R' \ . We shall use the collapsing theorem to formalize 
this argument. But before doiiijj su we note that a certain price has been 
paid for its intuitive simplicity; the get R is much higher than the set R', 
in faut| Jf | must be ®f the order of magnitude of £ |fl \ so that the best result 
to be obtained fiom the consi ruction is that the wider of ^ inu&t intrcase 
with| JT| like EogJ fl| - This fiives j weaker bound,. log|J?| compared with 
\R '■ '. if wp wish to estimate the order 

Connectivity on a T<midai Spaas \R\. Our earheBt attempts to prove 
that tf.™*™! has unbounded order led to the following curiftuft result: The 
predicate ^n^^, on an lit X G toroidalfy connected spneo \R\ has order 
5- n, The proof is hy construction: consider the space 




' re = 



in which the edges *,* and /, f are identified. Consider the family of subsets 
of if that satisfy the- cofiditjons; 

f ij All the shaded points belong to each A6-^ 

{a) For each X £ & and each (', cither both points marked <i :,i or both 
points £"■' are in -^, but no other combinations Are alio wed. 
Then it can be seen, far each X E -& t that X is either one connected figure 
Of X divides into two separate connected figures, Which case actuftllv 



2Ui 
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occurs depends only on the parky of \\ i\a ,J - ] £ X \\. Then using the Col- 
lapsing Theorem and the order (+*«,) t |J?| thyroid, wt find that ^, 
haH order |J?|/12. 

The- idea of this proof came from ihe attempt to reduce connectivity to 
non^y directly by representing the switching diagram: 




I f n.a. even number of sw i lehes are In die " down" position then a ia, oonn*«t«i 
fca a' end fi to 6\ If the number of down switches is odd, a is connected to 
4' and a' to t. This diagram can be drawn in the plane by bringing the 
vertical connections around the end- tbtn one finds, that the predicate 
|~a is connected to a' | his for order aDme constant multiple of | H\. ](we 
put the toroidal topology on R f the order becomes £ Constant times |fl| ; 
this ia aJao tru* for a 3-dimensions] non toroidal R. Because of these resuLts, 
we conclude that th* order -^ | R\ l - 1 obtained for <t> ammi is. too low, 

ArartEU in fbv<jy\ We have since shown that the order is at Least — | R\ ^ 
in tie pla,nt\ 

-Spme Other Geomtlrieol Prwiicatea. A number of other important geometric 
predicates that almost Certainly have unbounded aiders are: 

1. Symmetry: \~X is a symmetric ibout same line in the plane. I 13 

2, '"Twins": \X consols yf two disjoint congruent subfigures. - ] 
S. Concentricity! |~ X containa an Interior hoJe.~| 

CuriouflJy enoup;h h the predicate 

{~X haa a single connected comparent"| V \~X contains a hoLe~| 

has order 2. This can be shown l>v a construction using the Eulei relation, 
(Holes - 1 + Edges- Vertices,- Faces), even though each separately has, 
unbounded order- 

VII. CennecriviLv and Kciial cftitinnljukm. It seems intuitively clear that 
the reason that the ^bslrect quality of connectivity cannot be captured 
by a machine of finite order is, that it haa an inherently serial character; 
one cannot Conclude that a figure is connected by any simple order- 

" But, if s. pwrular uis line \i chtwHi in uilv*n« t llien wily outer 2 ii required! 
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independent combination of simple testa. The same is true for the much 

simpler property of parity. In the cast? of parity, there is a start* contrast 
between our "worst po&aibtft" result for finite-order machines (Jill) and 
the folding "best possible" result for the serial computation of parity. 
Let *i,jr s , ---,*., be any e«nmcrat]on of the polnta of R an d consider the 
following ulgcrithm. for determining the parity of |JC|: 

START: sett to 

EVEN: addltoi 

W=\R\ thenSTOP; parity is EVEN 

If i, - 0\ go to EVEN; otherwise go to ODDi 

Oi>l>. add 1 to j 

rf i - | J? | then HTOPi parity is ODD 

If x { = f\ go to ODL3; otherwise go to EVEN; 

where "go to <*" means continue tin- algorithm at the instruction whose 
name Is a. 

Now this program is "minimal" in two respects: first in the number of 
computation- steps per point, but more significant, in the fact that the 
program requires no temporary storage -place for partial information #e* 
Simulated during the computation, other than that required for the 
enumeration variable i. (In a sense, the process requires one binary-di^it 
of current mftiriiialion, but this, can be absorbed .am abovef into the 
algorithm- structure.) 

M his sunEPEte that it might be illuminating to ask for connectivity; 
how much storage is required by the best serial Algnnthm? The answer, as 
shown below, is that it requires no mo™ than about 2 time* that for storing 
the enumeration variable alone! To study this problem it seems that the 
Turing machine framework is the simplest and most natural, because of 
Its simple uniform way of handling information atorage. 

A Serial Algorithm fur ConntetitAly. Connectivity of a geometric figure 
X is nlwacterized hy the fact that bstwtrn any path lp,q) of points of 
X there i* a path that lies entirely in X. An equivalent definition, using 
the enumeration *,, ...,i Rj of the joints of R is; X is Connected if and 
only if for each point x f after the first point in X> there is a path to some 
Xj in X lor which j! > j". (Proof: by recursion* then, each prjjnt of X is con- 
nected lo the first point in X.) Using this definition of connectivity we win 
describe a beautiful algorithm to test whether X is connected, We will con- 
sider only figures that sue "reasonably regular"— 1« be precise, we suppose 
that X is bounded by a number of oriented, aimpLp, closed curves so that 
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for each point r, on a boundary there is defined a unique "next paint" 
tf on. that boundary,. We ehoose *,- to be the boundary point to the- kft of 
±j when facing the comntamenL of A. We will aLso asaurne that points x, 
.Hid .:;, lh.ii arf viaxtutiiv in • h l enumeratT n aic ij.V'Vi.-vii ,n iV. i'in.illy, 
we will assume that X does not touch the edges of the stmce R. 

START: Set i to and gr j to SEARCH 

SEARCH : Add 1 to i. If i - \Ii\> Stop and print "Jfis NULL. ' ' 
I£x,&X then go to SCAN , otherwise go to SEARCH . 

SCAN - Add 1 to L 1 f i = \ R | „ Stoo and print " X is nonruscted ." 

If j; .. t £ Jf or Jt, £ X go to SCAM , otherwise 
SetJ to p" and go to TRACE. 

TRACE; Set j to/* 

lij — i r Stop and print "X is diix&jmected-* 
Uj> j, (olli TRACK. 
[f/< I, go to SCAN. 

Notir* that at any point in the computation, it is necessary to keep track 
of the indexes of just the two points x. and x t . 

An&iym, SEARCH simpJy ftnda th* first point of X in the enumeration 
of R. Once such a point of X is found. KCAN searches through all of It, 
eventually testing every popnt of X. The current point, *, h of SCAN is 
tested tfs follow*: If *, h nol in X, then no t*st i* necessary and S^AN jjoe* 
on to j, , |. If the previous point r,_j was in X Land, by induction, is presumed 
to have passed the test! then x„ if in X\ is connected to X-i by adjacency. 
Finally, if JE.t X and *,_i^ Jf K then x, ia on a boundary curve H. TRACK 
circumnavigates this boundary curve. Now if B is a boundary curve it is 
either (i) an exterior boundary of a previously encountered component 
of X h in which cai« tome point of H must have heen encountered before 
or (ii> ii is an inteiior boundary cuTve r in which case a point of B must 
have been encountered before reaching pt l _ i which is inslite B or (iii) B is 
the e^terinr boundary curve (if a ]ieve-r-beP0rf--ejJcnimtftred component of 
A, the onJy case in which TRACE will return to x, without meeting an x. 
for which j" < i. Thus SCAN will run up to i - \R\ if und only if X has a 
single nonempty connected component. 

Note that we can (vuW the number of components of X by introducing 
K, initially zero, and adding 1 to K each time TRACE reaches the j = j 
eiit. Note also that the algorithm is quite efficient; the only points examined 
more them once are some of the boundary points, and none of them is 
examined more than three times 4&ee figure below), 
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2G<r 




|— ^^ Boundary pnLnU tynd 

by TRACE 
' 11 I a Emjudpry poifms mid 
by SCAM 

7V TUring A/o^j'kw VVrsjon af!tw Cuftnertivity Aigaritfim. It is convenient 
to assume that i? in a 2' X £" square ar™y. I.«t j,, . . ._ j^, l w ttn . enumera- 
tion of the points of A in the order 



1 



r+ i 



2-2" 



£2"- 1)2" +l 



2 n - 2\ 



This choice of dimension and enumtTjmon makes available a simple way 
to represent the situation to a Turing machine. The Turin K machine must 
he able t* anectfy a point x, of R, find whether a, C X r and in aisc *, is a 
bflnndfiry point of X, And the index j" of the "left neighbor" of je,. The 
Turing Machine tape will have the form 



<> 

1 I 


■ ■ n • ' 


I. 


i r ft i d 


^ 


- 1 ft ■ ' 


J, 


- ■ R ■ * 


K 



where ,L ' -a . ■" denotes an interval af n blank aqua***- Then the interval 
to the right of /, and J, can hold the j and y coordinates of a point of tf. 
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We will suppose that the Turing machine is coupled -with the outside 
*f«rld, i.e. t the figure X, through an "orade" that works- as follows: certain 
internal states of the machine have the property that when entered, the 
resulting neit state depends on whether the coordinates In the / [or J ) 
intervals designate a point in Ji. it can be verified* though the- details 
are tEdious, that all the operations described in the algorithm can be 
performed hy a fined Turing machine that uses no tape squares other than 
those in "- ■ tt ■ *' : intervals. For example, M J ' =■ \H\" if and only if there 
are aU Mifls in the. " - *n- -"a following; J, and l y "Add 1 to i" i» equivalent 
to: "start at J t and move Left, changing l's to O's unti] a is enKJuntered 
and changed to 1 or until I y is met. The only not! trivial operation is com- 
peting j* given /. But this requires only examining the neighbor* of JCj, 
And that Is done by adding -^ L r.n the J, end J y coordinates, and conauLtlng 
the oracle, 

Since the Turing machine can keep track of which "■ -ft- •" interval 
it it in, we realty need only one symbol for punctuation, so the Turing 
machine can be a 3-&ymbol machine- By using a block enraging, one can 
use a 2-aymbol machine, and, omitting details, we obtain the result- 

Theobew, For any t there is a 2-&ymbol Turing machine that can verify 
the connectivity af n figure X on any rectaftgiitar array R, uzinfi U-xx than 
{£+- *>loga|fi| squares af tape. 

For convexity there is a similar procedure that mjike* three tests; 

i. X is not disconnected by any vertical hne that does not. intersect X . 

Ji, The intersection of A with j«iy vertical line is a connected segment. 

iii. The oulor boundary of X does not change the sign of Its curvature. 

A detailed construction shows that each test requires roily one iiide* 
polftt, so that 

TliEOSEM. For any i there is a 2. symbol Turing machine that can verify 
the (.Qnwtity tf a. figure X on- any rectangular array It, Using less than 
{1 ■+- *} log* | fl \ squaTs of tape, 

'inis last resuk is certainly minimal since log, R squares are needed just 
to indicate a point of R t and all points, must 1m.; ^xjimincd, We are quite 
sure that the connectivity algorithm is minimal, also. In its u&e of tape, 
but wb have no proof. In Fact, we do not knew miv method, in general, 
to show thflrt an algorithm is minimal in storage, esoept when information- 
theoretic arguments can he used. Incidentally, it is noL hard to show that 
H^l « prime"] requires no more than {£ + «) log 3 | ft| squares 4 and pre- 
sumably n^risfi mint than (2 - t> log 3 | R\ }. 

We do not definitely kimw ?jiy geometric predicates that require higher 
orders of storage, but we suspect that in an appropriate sense, the topological 
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Wiuavalenee of two ftRuns [e.g., two Com rj-^nimU of X) rt-quires something 
mor* like |fl| than Kite lOjg|i?| squares. There are, of course, recursive 
function-trL*orOtic predkatea that require arbitrarily high, indeed non- 
wmou table, Drdeia of storage, but none of these is known to have straight- 
Jorward geometric interpretations. 

VIII. Mulei-la>er perceptions. W* have found a number of limitations 
of perceptions, as defined above, and wl' have .-suggested (hat these may 
point Mwwrd as yet unknown theorems ahuul parallel machines in general. 
On the nther hand one su*p*oti that some, a1 least, of the result* above 
are not bo gen*™ |, and might not survive minor J^jsuliona of the definitions. 
One direction of v^neraliaatlon thai seems imporljitii is that of relaxing 
the constraint that the #'* be simply weighted and added, Wo have not 
found any particularly enlightening generalization on the lowest level— e.g., 
i if replacing addition by an arbilniry commutative operation. An easier 
direction ti to consider compositions of perceptions. The remainder of 
this section eKploreii- some kinds of composite perceptions. LfnforLwnately 
we do not understand them very well, so tbia section is more concerned 
with probl*m- posing than with problem- solving. 

Ckirnba Afacfliufls, Consider functions- of th* form 

r z*r r z»* *. > *r\ > t~\ . 

This form was proposed and rc-ahzed in a scries of machines built hy A. 
Camba [2|. Jt is essentially an order-] composition of order- 1 perceptrona, 

and i« of interest to u* for a number of reasona: 
{i) The parity problem is solved neatly by 



rj2<-«T2;*>n>o-i. 



.!-:■ jij 



Thus only \R\ functions are needed, each itself of order 1 in the \x,\. In 
fact, any predicate f(\X\ ) that depends- only ofl the area|X| can be 
ionised, as 

The problem that Jed 10 our formulation of the AND -OR theorem «Lso 
ie solved neatly: 

rr z*-z*>«i + rz*- z*>«~U2i 

Is 1 if and only if \Xf\A\ >\XftC\ and \XnB\ > \ Xn C\, This 

might suggest that thta class of machines might tran^&nd the other kind* 



■. 
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of limitations we hove found for machines of finite- airier. • 

We art quite certain that this impression is misleading; that the deeper 
geometric properties am; still oucskte the reach of this kind of "2-layer" 
perceptron. The iji«l U9 ian uf AND and OR is due to the E-bm T construction; 
any Boolean function Ja obtainable, in such a manner, through its normal 
form, but for moat function* there will stilt b« too many terma for practical 
interest. The fiXn A\ > \XC\ C\ -| type of predicates are within reach 
because they are simple area functions And hence fit. precisely th* Lnm-r £>. 
nraL-ftrniT predicate forme. In fact, ony classJ^of figure can be recoflttssed 
by a Gamba -machine because 

rr rz^kiJtn ><n 

realizes. Lt. Rut, this general form requires a special "Gamta-mask" for 
each XGJr. Although the above examples ahow that in special casus 
more economical representations arc possible, this is not true in genera] 
fas one can see by considering the number 2*" of possible functions). In 
parLLCular we conjecture, for esattiple, that for the Connectivity Predical*. 
the machine would require a number of masks of an order approaching 
the number of simple-closed -curves in R, Even for convexity, we doubt 
thai that predicate can be realized with significantly fewer than the number 
of *'s needed for th* order-3 1 -layer machine. 

(iii 3:i spite oi its apparent simplicity, analysis of the geomeLrie predicate 
problem for (iamba machines appears to require methods cjuite different 
from these we have used. First, because of the arbitrary order 1" predicate 
permitted 1 In the inner sum. the notion of order dues not seem uj apply, 
and thenreknE must concern restrictions n n the number* of terms. Second, 
we have not found a way to Carry the group-averaging methods into the 
inner a u coefficient*, so that we Cannot uae the Icebniquea that Come from 
the group- invariance theorem. It is difficult to see how to analyse other 
multi-layer and composite perceptions until this simple case ia better 
understuud How much weaker are the machine* with *„ > £> or those 
with all *= 07 We have no characterization of what they can do. 

(iii] The Gam bA- machine is of considerable practical interest because 
of the possibility of realising the inner, and even the outer, sums by in- 
expensive, highly parallel optical method*.. IJsjng coherent light and properly 



' N"<H* thai the GarohB-HMtfhuie con hnvn finlnr an Injur a> |rt , in lh#< J£,-|, If trie 
jtndLcDU- th,r*5h(±d ««* jemnvtwl ih€ti ( because 

J i 1 \ i 1 

ntw? *wjld have nvrr*l> *n vrder-t function, tti Uic firjf. 
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prep^ed photographic transparencies, one can realize each inner sum (even 
with complex co*fflclenta!) with a picture p., whose density at point x, is « 4 , 
Ry shrewd optks, on* can even do this (with fixed p,) for aU transitions 
nf the StnyrCe pattern X. Because of these technological possibilities it Ls 
important to have a better theory' we esipea thai the resuLt will be favorable 
to problems like recogn i tion of punted characters* hut still vtry pour for 
the more abstract properties like detection of cotmcctivity,, symmetry, 
topological equivalence, and the like. 

IX. The diameter- limited pcrceptroii. In thi& section we discuss the power 
3-nd limitations of the ''diameter-limited" jjerccptrons: thoae in -which 
each + pun see only a circum±st-rihcd portion of the retina ft. 

We enrssuier a machine that sums the weighted evidence about a picture 
obtained hy eiperimenls *, each of which report on the state of affairs 
within u circumscribed region r, of diameter less th&ti or equal to some length 
D. That is, Diameter (S{*)3 < I). We will suppose that D is uniform over 
thi- s'a of the machine (each actual region that affects a #, can he smaller, 
hut not larjfer}. We suppose also that in a practical sense D is smaLl com- 
pared with the full dimensions of the space fl, That ia. /J should be small 
enough that none of the *'& can see the whole of an interesting figure (or 
«Ue T* e would not have <in effective limited -diameter situation, and there 
would he no interesting- theory J hut D should be large enough that a #, has 
* chance to detect art interesting- "local future" oJ' the figure. 

We will consider drat some things that a dimeter-limited pereeptron 
can recognize, And then some of the things it cannot. 

in) Blank picture, or bkttk picture. A diameter-limited pereeptron can 
tell when a picture is entirely Mack, or entirely white; suppose that 1he 
set of $,'* is chosen to cover the. retina in regions, that may overlap, and 
I h<it we define p, to be wro when aLL the points it can set are white, other- 
wise its value is 1. Thtii^ft > if the picture has one or mum Miick points, 
and not if the picture Ls blank- Similarly, we could define the $,'* to be- 1 
when they see any white point, otherwise, thus di? I h^u ishing- the all- 
black picture from all nlhers. 

For Lutcr examples, it is important here to notice why these y*Ucrns 
can be recognized; it is not that any tf-unit can really say that there is 
strong evidence that the figure is all-white ij although it has a slight cor- 
relation with this}; but any $ can definitely say that it ban conclusive 
evidence that the picture Ls rail a]] while. Some interesting paliMiit h;i\i- 
Lhis character' that one can reject all pictures not in the class beeause each 
must have, somewhere or other, a local feature that ts deSnitive and can 
be detected by what happen* within a region of diameter D. 

(b) Area cats. We can distinguish, for any number S, the class of figures 
whose area is greater than 5. To do this we define a *,- for each point to 
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be J if that point it black, otherwi.se. Then 1] *, > <? i* B recognizer for 
the class in queslion. <Qne «m do aligJatly better; if the p'* look at regions 
Of him A, then <]ii e can recogime this pattern by using only of the order 
of (fllogAk-.4 united 

(c) NoninteneciiTig lints. One cast say than a pattern is composed of 
iicmintetsectbig lines if, in each small region, the pattern is competed of 
separate LLne-s*gments, or blank. Then, if wo make each p have value aero 
when this condition is met, unity when it is nut, then J£*, > Q will reject 
all figures not in the class. 

(d| Triangles. We can make a diameter-limited perceptron recognize 
the figures conststing of eiactly one triangle (cither solid or outline) by 
the following trickr We uu two kind* of tf h s; the first has weight + l if 
it* field contains 3 vertex (two line segment* meeting; at an angle), other- 
wise m value is zero. The second kind, *,*. has value zero if its field is 
blank, or coiiiains a lme segment, solid black area, or a vertex, hut has 
value - 1 if the field contains anything else, including the end of a line 
segment. Provide enough yf these #*e &> that the entire retina is Covered, 
ffl nonov-MlBpping fashion, hy both types. Finally assign weight 1 to the 
first tyjw and a very Lar K e positive weight W to those of the second type 
Then 

2>, IV».'<4 
will be a specific recognizer for triangles. [But sl&y the null picture is 
accepted,) Similarly, by requiring that the first kind of unit recognize right- 
angle vertices, the machine can he made to recognize the das* of rectangle* 
(setting the threshold to be <5). 

Note that this does not generalise to a very wide class of geometric 
recognition abilities. The triangle and rectangle case* Arc rather peculiar 
the triangle because it is the simplest figure that has true vertices. The 
rectangle can be recognized because it has four equal angles; ttu system 
Cannot he ^penalized to recognize, for example, e**etly the equina It 
ia interesting, in view of the limitations we will establish shortly, to see 
why these patterns, can be recognised by the diameter-limited machine; 
a rectangle is l.he only figure that has four w fewer right angles and no 
free Line «nds r etc. 

(e) Absolute ttmplal* -m&dung. Suppose that one wants the machine tn 
recognize exactly a certain figure J( and no other, Then the diauielcr- 
Limted machine can be made to do this hy partitioning the retina into 
rrflkms, ami En each region a £- function has a valoc if that part of the 
retina is exactly matched to the corresponding part of X Ui otherwise the 
value is 1. Then 

if and only if the picture is eiactly X$. 
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Note hcwever, that tbi« scheme work* just yn a partkiiJar hj«t in a 
particular position. Jt cannot he W nwaJL!*«L id «c;n K ni» a particular object 
m any portion tor ev*n, in gene,,], j„ two poaLLE(liwJt ]n fjlct m ^^ 
m the next section l,bat eve* the simplest possible figute, namely „ne that 
coflftjstaof jort one print, ca»*.ot be recognized independently of potion! 

W (&■««,?. The remarks Ln fa. Example e, footnote apply to the 
diameter- limited case. 

Wfti/ioh* of Diamtor.lwitel Perwpt™. Now we corner tomfi „/ 
the base Limitation* of the diameter-limited perception, by exhibitms and 
analysing aomv patterns they cannot recognize. 

(S) m few containing w Hugh black point. This is the fundament*! 
CGUnter-eijimple. We want a machine 

W accept flfcurea with area I, but reject figure* ™th area or art* gr M tar 
than 1., Clearly thia can he defined by t»„ area cfcts lie , a™ > AND 
area <2), but it cannot be realized by a linear threshold function with 
In* area- restriction. 

To see that this can^t be dun,, suppose that [*f + ]«,] And* hm b«n 
Minted. Prc«nl fii,t the bknk picture. X,. Then, defining /(*, = T.+.Uy 
we have ft¥J ^ *. Now present . figure, X lt containing only me ^ - 
We must then have 

fix,) £ B, 

The change in the uim mint he doe to a change in the value* of some of 
the* *. Tn fact, it must be due to changes only in tfa for which x^SM 
since nothing ebe [n the picture hat changed. In any cast.. 

Now choose another point Xj *hich in father than D awiy from ^ Theri 
no 5(#) can contain both ft, and jf a . For the figure A", containing only a a 
we must iiJso have 

Now consider the figure X i2 containing both *, and x,. The addition to 
*i. of the point ft, can affect ttttly *'a fur *-hkh i€ SW, and the^ ar e 
changed exactly „ they are changed when the alL- blank picture X t . i% 
Changed to the picture X v . Therefore 

ftXiA-fiXJ + ifVCJ-fiXl} 
and by (1) and (t) t 

ftXJ > f. 
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but we require that 
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f(X l? ) £ U. 



REMAhK. Of couree., this is the same phenomenon, noted tn the introduc- 
tion to {II. 

(hi Art® a^m^nis. The diamet*r-limit«t penxplron cannot recognize 
the claes. of figure* wh<w urcae /i lie between two bounds A, i 4 i i4 a . 

Phuuk. This foLLows fwm the method of (a> above, which is a special 
Case of this, with 4,- 1 and.A t = I, But using- the method of §1, tiiample 
jviij, this recognition is possible with order 2 if the diameter-limitation 
is reLaied. 

(i) OwiJMStodiwsj. The diameter-ltmited perception cannot decide when 
the picture is s single,. Connected whole, aR flis.tiMjji.ji*Jjed from two or moie 
disconnected pieces. 

Proof, Consider the- four pictures 
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and suppKMB that ihe diameter D "a of the order Indicated, by the doited 
circle. Now figures A n , and A K , are wnnwled. but X w and X„ are dis- 
connected. Suppose th«t there weie a seL of £"s and n's and a such that 

so thiil these four figures wert correctly separated. But then,, just as in 
the previous argument W* would have for all #„ 

*.(Jtn) - » I (-*!(* + *,UC m ) - *,(Jt„) 

because the twn changing regions are more than D apar^ hence 

2*i#i-(J£ u } a * + #-#-* 

contradicting the- separation, requirement. 
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